This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the appUcant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□I BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHitE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



UK Patent Application „.,GB .,,,2 333 656 .,3, A 

(43) Date of A PubUcdtion 28.07.1999. 



(21) 


Application No 9S01 380.8 


(51) 


INTCL^ ■ 
H04N 7/50 


(22) 


Date 0' r-iiing zz.ui.isnni 










(52) 


UK CL (Edition O ) 

H4F FD10 FD12M FD12X FD3D FD3M FD3P FD3T 






(71) 


Appiicantis) 

British Broadcasting Corporation 




FD30E FD30K FD30R F032 FRW 




(Incorporated in the United Kingdom) 


(56) 


Documents Cited 




Broadcasting House/ LONDON. W1A 1AA. 






United Kingdom 








Inventor(s) 

Peter John Brightwell 
Nicholas Dominic Wells 
Philip Nicholas Tudor 


(58) 


Field of Search 


(72) 




UK CL (EdHion P ) H4F FRO FRD FRG FRM FRP FRR 
FRS FRT FRW FRX 

INT CL^ H04N 5/00 5/76 5/91 5/917 5/92 5/926 5/93 
7/00 7/12 7/24 7/26 7/30 7/32 7/34 7/36 7/48 7/50 




Agent and/or Address for Service 




7/52 9/00 9/79 9/797 9/80 9/804 9/808 9/87 11/00 


(74) 




11/02 11/04 




Mathys & Squire 




OnlineiWPI 




100 Grays Inn Road, LONDON, WC1X 8AU 






United Kingdom 







(54) Abstract Title 

Video compression 



(57) An MPEG2 decoded video signal is accompanied by a representation of the coding decisions to aid 
downstream re-encoding. The representation is MPEG compliant bit modified to reduce the number of bits. 
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COMPRESSED SIGNALS 



2333656 



This invention concerns the handling of compressed or bit-rate 
reduced signals, 

5 ■ In the most important example, this invention relates to the MPEG2 

video compression system and the following description will take MPEG2 as 
an illustrative compression (or bit-rate, reduction) system. 

As compressed video increasingly find its way into the programme 
chain, it becomes more likely that bitstreams will be decoded and re-encoded 

10 Examples of where this happens include switching between different 

bitstreams and changing between bit-rates and compression formats. It is 
important that unnecessary loss of quality is avoided in these cases. 

There is proposed in EP-A-0 766 576 a solution in which an 
information bus accompanies a decoded signal and is available in a 

15 subsequent encoding process. In apparatus and channels designed for this 
purpose, it is possible for the decoded, picture signal to be accompanied by 
an information bus which contains all the side information which is contained 
in the MPEG signal, alongside the DCT coefficients. The information bus 
may also contain additional information which is related to the control of the 

20 coding or decoding process but which does not form part of the coded 
bitstream. It can include, for example, statistical information, information 
concerning errors together with instructions for use in error concealment, 
candidate motion vectors or recommended quantizer step sizes. 

There are disclosed in co-pending international patent application 

25 PCT/GB97/01862, techniques by which an information bus can pass along a 
video pathway. One method uses an information bus embedded in the 
lowest bits of the decoded video signal (in ITU-R Rec. 656 format), such that 
the picture quality is not visibly affected. In one version of this, the video bits 
used to convey a particular coding decision are spatially and temporally 

30 locked to the area of the picture to which it corresponds. Alternatively, the 
information may take a different form, for instance using non-active parts of 



the picture, such as the vertical blanking inten/al (VBI). or by using a 

separate channel. 

A problem arises where it is not possible to embed the infonnation 
bus in the video signal without suffering excessive data loss and VArfiere.the 
equipment concerned has not been designed with an appropriate channel to 
carry the information bus separately. An example of this is when the video 
signal is recorded by a digital VTR that employs DCT-based compression, 
which tends to change the lowest bits of the signal. 

Accordingly, the present invention consists in one aspect in a video 
compression system in which an encoder operates on a video signal and 
takes coding decisions in a compression encoding process to generate an 
encoded signal and in which a decoder operates on the encoded signal in a 
compression decoding process to generate an decoded signal, wherein a 
representation of the coding decisions or some of them is provided in bit-rate 
reduced form and accompanies the decoded signal through a bit-rate 
reduced channel. 

The bit-rate reduced channel may take a variety of forms including one 
or more audio channels (such as VT audio tracks or a separate audio 
recorder); unused VBI lines, a video channel with significant error correction 
and appropriate data channels (including DVCpro data channel). 

For the case of an MPEG-2 bitstream, the coding decisions typically 
include: 

• Picture dimensions 

• Frame rate 

• Picture structure (frame-coded or field-coded) 

• Picture type (I, P or B) 

• Whether macroblocks intra-coded or use prediction 

• Whether forward, backward or bi-directional prediction is used 

• Whether frame-based or field-based prediction is used 

• Motion vectors 

• Quantiser visibility weighting matrices 

• Quantiser step 



• Buffer state of a decoder that is decoding the bitstream 

• Information on bit-rate usage within the bitstream 

When it is possible to keep all (or a sufficient sub-set of) the decisions 
the same on recoding. it can be shown that the switch is transparent In other 
cases, such as when ia new bitrate is used on recoding, it is still 
advantageous to reuse many decisions (e.g. picture type and motion vectors), 
and take account of others (e.g. quantiser scale). 

In one form of the invention, the bit-rate reduced coding decisions 
pass through a convenient separate channel such as one or more audio 
channels. An alternative is to use the bit-rate reduction of the coding 
decisions to enable them to be embedded in the video signal in a more 
robust manner, such that they can be recovered without errors. 

Where the chosen channel has insufficient capacity to support 
sending the coding decisions with fixed length codewords, variable length 
coding can be used. However, use of variable length coding means that the 
short-term bit-rate can exceed that which fixed-length coding would give, and 
some method of reducing this is required. 

This invention, in one aspect, provides for decisions to be modified to 
reduce their bit-rate whilst signaling to the recoder that the decisions have 
been changed and so no longer correspond to the original bitstream. 

Another aspect of this invention addresses the problem that where an 
infonmation bus is embedded in a video signal, it is not possible to detect 
where parts of the picture have been changed by caption insertion, 
cross-fades, and the like. 

Another embodiment of this invention can be used for converting 
between an information bus embedded in the video signal and a 
side-channel based information bus. This would be typically used for 
integrating a DCT-based VTR into an ITU-R Rec. 656 infrastructure. 

On embodiment of this invention consists in a system for producing a 
bit-rate reduced representation of coding decisions suitable for passing 
through a channel (comprised of e.g. one or more audio channels) in which 
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the available bit-rate may be insufficient to carry the coding decisions 
themselves. Preferably, variable length coding of the information is 
employed. 

The following are examples of this aspect of the invention:- 
5 . A. An MPEG-compliant video bitstream, with "dummy" 

coefficients and coded block patterns inserted to keep the bitstream legal. 
These would not represent the coefficients and coded block patterns used for 
the original bitstream, but would give the shortest legal codewords. 

B. A bitstream that uses a modified version of the MPEG 
10 format in which the macroblock syntactical element is altered so that it no 

longer includes coefficients or a coded block pattern. This would mean a 
small change in the way in which the bitstream is decoded. 

C. As form B. but in addition, some of the coding decisions, in 
particular the motion vectors, are coded in a non-compliant way in order to 

15 reduce their bit-rate. Depending on how this is done (methods for this are 

described below), there may be a small or a large change to the way in which 
the bitstream is decoded. 

The invention will now be described by way of example with 
reference to the accompanying drawings, in which:- 

20 

Figure 1 shows a system according to this invention in which an 
MPEG bitstream is decoded for passage through a video channel to a 
encoder, with coding decisions being taken from the decoder and bit-rate 
reduced for passage through an information stream channel which 
25 accompanies the video channel to the encoder; 

Figure 2 illustrates a modification in which coding decisions are 
extracted from the MPEG bitstream, in front of the decoder; and 

30 Figure 3 shows a system according to the invention in which coding 

decisions embedded in a video signal (in accordance with co-pending 
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international patent application PCT/GB97/01862) are extracted before the 
video signal passes through - for example - a digital VTR. 

Referring initially to Figure 1 , this shows a system in which an MPEG 
5 bitstream is decoded in MPEG-2 decoder (10) to produce a Rec. 656 video 
signal and a separate output of coding decisions, that are extracted from 
the bitstream. The latter would be typically take the form of an "Information 
Bus" as described in EP-A-0 766 576. The coding decisions, in particular 
the motion vectors, if necessary are then modified (as described below) in 

10 the modify decisions block (12) to reduce the number of bits required to 
code them. They are then variable length coded in VLC (14) using one of 
the above forms. Additional information can be added through to signal 
whether modification was performed. The resulting bitstream is then 
buffered in buffer (18) to form an Information Stream. This is then adapted 

15 to the particular channel (for instance it may be multiplexed between a 

number of audio bitstreams which are sent to an audio recorder). The level 
of the buffer (18) is sent back to block (12) to control the modification 
process and hence the short-term bit-rate entering the buffer. 

After passing through the Information Stream channel (20) and 

20 subsequent buffering in buffer (24), the Information Stream passes to a 
VLD (26). The Information Stream channel (20) accompanies the video 
channel (22) which would typically include equipment such as a VTR. The 
decoded Information Stream can then be used by an MPEG encoder (28) 
whose input is the output of the video channel. Where the Information 

25 Stream contains information signalling that the coding decisions have been 
modified, the encoder may then remake some or all of the decisions. 

There a number of possible options for modifying the coding decisions 
to reduce the number of bits required to code them: 

30 1 . Macroblocks can have up to four motion vectors, depending on the type of 
prediction and motion compensation employed. Fewer, or no, motion 
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vectors can be sent (this will require changes to other MPEG syntactical 
elements: macroblock_type and motion_type). 

2. Motion vectors are coded differentially in MPEG, using a set of predictors 
that can be updated on a macroblock rate. The number of bits required to 
code a given motion vector increases with the size of the corresponding 
differential. Therefore motion vectors with large differentials can be 
modified to have a smaller differential, including the case of zero 
differential, 

3. The MPEG syntax contains picture rate elements that specify the 
horizontal and vertical ranges of motion vectors that will appear in the 
forthcoming picture (the f__code values). The value of these affect how 
many bits are used to code the (differential) motion vectors. By 
employing options 1 and/or 2, it may be possible to modify the f_code 
values and code the remaining motion vectors in fewer bits. 

4. There is an overhead (6 bits) where the quantisation step parameter 
changes between adjacent macrobiocks. By changing the quantisation 
step parameters to be the same, this overhead is removed. 

In addition there are refinements concerning when to employ the above 
techniques, in order to minimise the number of macrobiocks that must be 
modified to save a given number of bits in the Information Stream. The 
purpose of doing this is to maximise the amount of video signal that can be 
receded transparently: 

1. B-pictures can use bi-directional prediction and so tend to require more 
bits for their coding decisions, especially motion vector bits. Because of 
this, and also because any loss in transparency on receding is not 
propagated to other pictures, macrobiocks in B-pictures should typically 
be modified with higher priority than those in other types of picture. 



- 7 - 



2. Macroblocks around the edges of the picture should have higher priority 
than those nearer the centre, because loss of transparency is less likely 

• . to be noticed around the edges. 

5 ■ 

3. By "looking ahead'' a nunnber of macroblocks. it may be possible to "spot" 
sequences of macroblocks that would give a good bit-rate saving. 

When modification is performed; one or more of the following methods can 
10 be used of conveying this information: 

1 . One or more new syntactical elements are added to the bitstream. This 
method is not possible for form A of the Information Stream (the 
MPEG-compliant form). 

15 

2. Unused values of certain syntactical elements can be used, in particular, 
removing the coefficients from the syntax removes the need the "Not 
coded" values of macroblock__type. These can be reused to indicate that 
the decisions have been modified. This method is not possible for form A 

20 of the Information Stream. 

3. For form A only, the information can be conveyed in the choice of the 
dummy values of the coefficients and/or coded block patterns inserted to 
keep the Information Stream as a legal MPEG bitstream. 

25 

4. User data is used. 

For form C of the Information Stream, non-compliant coding is used to give 
greater reduction in bit-rate. Examples of methods to do this include: 

30 

1 A more complex motion vector predictor is used, for instance a two 
dimensional predictor that takes account of the vectors used in the 
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macroblocks above the current one as well as the macroblock to the 
left of it. 



2. A model is calculated for the picture, or a portion of it, that provides a 
simplified description of a coding decision. Typically user data v/ould 
be used to send the mode! at the start of the picture. Macroblock-rate 
information is used to describe deviations from this model. For 
motion vectors, the model can be considered as an advanced 
predictor. 



10 



3. Techniques such as quadtree coding are suitable for decisions that 
only have a relatively small number of possible values, such as 
macroblock type, motion type and DCT type. Quadtree coding can 
also be used to code vectors that appear in a "menu" - and 
15 reference is directed in this regard to GB 2240005. 



All the above methods can be extended to allow modification of the 
coding decisions where necessary, and again any modifications could be 

20 signalled in the Information Stream. 

An enhancement to the system described here is shown by the 
dotted lines in Fig 1. Information is added to the Information Stream, 
typically as user data, that gives information about the decoded video 
signal. When the Infonnation Stream is decoded, the information can be 

25 checked against the video to be receded. This will enable the recoder to 
make new coding decisions where the video signal has been changed 
substantially, for instance where captions have been added. Typically the 
information would be a macroblock-rate checksum (calculated in block (16) 
and compared in block (30) ) if the video channel is transparent, or a 

30 macroblock-rate measure of mean and variance, if the video channel is not 
transparent (e.g. when it passes through a VTR). Statistical measures other 
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than mean and variance can be employed, and can be calculated on picture 
regions other than macroblocks. 

. A variant of the inv.ention is shown in Fig 2. where blocks serving the 
same function as blocks in Fig 1 are given the same reference numeral and 
5 - not described further. In Fig 2, rather than coding the information Stream • 
from the decisions extracted from the bitstream.. the MPEG codewords 
representing the decisions are stripped out and passed to block (50) with 
changes made as required where there is a need to modify decisions or 
.add additional information. The "modify bitstream" block (50) thus serves an 

10 analogous function to "modify decisions" block (12). although operating on 
variable length coded representations of the coding decisions, rather than 
the coding decisions themselves. 

Another variant is shown in Fig 3. Here, an Information Stream that is 
embedded in a video signal ( as mentioned above and as described in 

15 PCT/GB97/01862) is converted into one of the formats described, and back 
again. This is suitable for adapting a VTR etc. inio an infrastructure which 
otherwise employs an embedded Information Stream. More specifically, 
Fig 3 shows a block (60) which senses to extract an embedded information 
bus from an incoming video signal. The extracted information bus is 

20 modified and variable length coded in the same manner as shown in Fig 1. 
to pass through an Information Stream channel 20. Downstream of the 
restricted video channel 22, the Information Stream is variable length coded 
for re-embedding in the video signal at block 62. 



1 . A Video compression system in which an encoder operates oh a video 
signal and takes. coding decisions in a compression encoding process to 
generate an encoded signal and in which a decoder operates on the 
encoded signal in a compression decoding process to generate an 
decoded signal, wherein a representation of coding decisions is provided 
in bit-rate reduced form and accompanies the decoded signal through a 
bit-rate reduced channel. 

2. A video compression system according to Claim 1. wherein the bit-rate 
reduced representation of the coding decisions is passed through one or 
more audio channels accompanying a video channel carrying the 
decoded video signal. 

3. A video compression system according to Claim 1, wherein bit-rate 
reduction of the representation of the coding decisions, comprises 
variable length coding. 

4. A video compression system according to Claim 1. wherein short term 
control over bit-rate is achieved through modification of the coding 
decisions. 

5. A video compression system according to Claim 1. wherein the bit-rate 
reduced representation of the coding decisions comprises an 
MPEG-compliant video bitstream. 

6. A video compression system according to Claim 5. wherein the bit-rate 
reduced representation of the coding decisions comprises an 
MPEG-compliant video bitstream in which the coefficients and coded 
block patterns used for the original biistream are replaced by with dumm 
coefficients and coded block patterns. 
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7. A video compression system according to Claim 1. wherein the bit-rate 
reduced representation of the coding decisions cornprises an 
non-compliant MPEG video bitstream in which the coefficients and coded • 

' 5 block patterns used for the original bitstream are deleted with alteration of 
the macroblock syntactical element. 

8. A video compression system according to Claim 4. wherein modification 
of coding decisions comprises modification of motion vectors. 

10 

9. A video compression system according to Claim 8. wherein modification 
of coding decisions comprises modification of motion vectors where the 
differential is greater than a certain size. 

15 10. A video compression system according to Claim 4, wherein modification 
of coding decisions includes looking ahead to find groups of macroblocks 
to be modified. 

11. A video compression system according to Claim 4, wherein modification 
20 of coding decisions consists in preferential modification of B-frames. 

12. A video compression system according to Claim 4, wherein modification 
of coding decisions consists in preferential modification around the edges 
of the picture. 

25 

13. A video compression system according to any one of the preceding 
claims, in which information about the decoded video signal is carried in 
said representation to enable detection of changes in the decoded video is 

30 changed. 
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14. A video compression system according to Claim 13, wherein said 

information about the decoded video signal comprises a check sum of a 
macroblock or other picture region. 

5 15. A video compression system according to Claim 13, wherein said 
information about the decoded video signal comprises statistical 
information. 

16. A video compression system according to Claim 15, wherein said 
10 information comprises the mean and variance of a macroblock or other 

picture region. 
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